#CT/QB3 9^4 3 6 




Office 



09/806702 

PRIORITY 
DOCUMENT 

SUBMITTED OR TRANSMITTED IN 
COMPLIANCE WITH RULE 17.1(a) OR (b) 



INVESTOR IN PEOPLE 




The Patent Office 
Concept House 
Cardiff Road 
Newport 
South Wales 
NPIO 8QQ 



I, the undersigned, being an officer duly authorised in accordance with Section 74(1) and (4) 
of the Deregulation & Contracting Out Act 1994, to sign and issue certificates on behalf of the 
Comptroller-General, hereby certify that annexed hereto is a true copy of the documents as 
originally filed in connection with the patent application identified therein. 



In accordance with the Patents (Companies Re-registration) Rules 1982, if a company named 
in this certificate and any accompanying documents has re-registered under the Companies Act 
1980 with the same name as that with which it was registered immediately before re- 
registration save for the substitution as, or inclusion as, the last part of the name of the words 
"public limited company" or their equivalents in Welsh, references to the name of the company 
in this certificate and any accompanying documents shall be treated as references to the name 
with which it is so re-registered. 



In accordance with the rules, the words "public limited company" may be replaced by p. I.e., 
pic, P.L.C. or PLC. 



Re-registration under the Companies Act does not constimte a new legal entity but merely 
subjects the company to certain additional company law rules. 



Signed 
Dated 



14 JAN 2000 



An Executive Agency of the Department of Trade and Industry 




is Page Blank (uspto) 




nt'^'^inn 1/77 



Its Act 1977 
(Rule 16) 



Request for grant of a patent 



Patent 
Office 




1/77 



The Patent Office 

Cardiff Road 
Newport 
GwentNP9 IRH 



1 . Your reference 



03 32699 



0^N0V9Q zmosd 1 



2. Patent application number 



0 3 NOV 1998 



3. Full name, address and post code of the or 
each applicant 



POl/7700 0.00 - 9824033.6 



9824033.6 



BRITISH TELECOMMUNICATION^ puDiic iimiiea company 
81 Newgate Street 
London EC1A7/U 



Patents ADP number 



If the applicant is a corporate body, give the 
country/state of its incorporation 



GB 



4. Title of the invention 



INTELLIGENT ASSISTANT DEVICE 



5. Name of your agent 



VENNER, SHIPLEY & CO 



"Address for service" in the United Kingdom 
to which all correspondence should be sent 



20 LITTLE BRITAIN 

LONDON 

EC1A 7DH 



Patents ADP 



1669004 



If you are declaring priority from one or more 
earlier patent applications, give the country and 
the date of filing of the or each of these earlier 
applications and the or each application number 



Country 



Priorit^^^lication number 



Date of filing 



7. If this application is divided or otherwise Number of cariier application Date of Filing 

derived from an earlier UK application, 
give the number and filing date of the 
earlier application 



8. Is a statement of inventorship and of right YES 
to grant of a patent required in support of 
this request? 



Patents Fqrm 1/77 



9. 



U. 



Enter the number of sheets for any of the 
following items you are filing with this fonn. 
Do not count copies of the same document 





state how many against each item. 

Priority documents 
Translations of priority documents 
Statement of inventorship and right to grant 
of a patent (Patents Form 7/77) 
Request for preliminary examination and 
search (Patents Form 9/77) 
Request for substantive examination 
(Patents Form 10/77) 
Any other documents 



12. Name and daytime telephone number of 
person to contact in the United Kingdom 



lAVe request the grant of a patent on the basis of this application. 
Signature ^^j^ 

MATTHEW READ 
0171-600 4212 



- 1 - 



Intelligent assistant device 
Description 

The invention relates to improvements in or relating to or for an intelligent assistant 
device for assisting in the management of information flows for a user. 

Modern computers are multifunctional devices which enable a user to process data 
not only from a local source but also remote sources connected through local area 
networks and wide area networks such as the internet. The user can send and 
receive e-mails and generally use the computer as a workstation from which many 
different day-to-day tasks can be performed. 

So-called personal organiser software is available to facilitate management of such 
day-to-day tasks. For example Microsoft Outlook marketed by Microsoft 
Corporation, provides a group. of program suites accessible through a single window 
which allows e-mail to be sent, logged, received, making user of address information 
from a common database. Also, the software includes a diary and scheduling 
system which allows the user to interrogate diaries of other users on a network, so 
as to arrange meetings and other appointments. However, with this package and 
others, the user must make a large number of decisions concerning prioritisation 
and scheduhng of incoming and outgoing data. 

In accordance with the present invention, intelligent reasoning computing 
techniques are used in order to improve functionality. 

In accordance with the present invention there is provided an intelligent assistant 
device for assisting in the management of information flows for a user, comprising: a 
plurality of information processing modules to process individual types of information 
flows for the user using intelligent reasoning computing techniques, and a co-ordinator 
module to co-ordinate operation of the information processing modules, the co-ordinator 
including a library of plans corresponding to different tasks to be performed, a scheduler 
to schedule activities of the processing modules to perform the tasks, and an executor 
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^sponsive to the scheduler to instrua the modules to parfonn the tasks according to one 
of the plans. 

In accordance with the present invention there is also provided a dia^ module for an 
^ mtelhgent assistant device, including means to define a plurdity of timeslots. input means 
to provide a plurality of predetermined fuzzy definitions of a desired time for a diao^ entn., 
means definmg a relationship for each of the fuzzy definitions corresponding to timeslots ' 
for the enay ranked in an order of preference, and processing means responsive to a diary 
entry from the mput means to review the timeslots and allocate the entry in accordance 
0 with the ranked order of preference. 

In accordance with the present invention there is also provided a communications 
organrser for communications made over a network, comprising means for monitoring 
communrcatrons received by a user together with details of the sender, learxUng means to 
learn the user's preference for communications from particular senders, and means for 
mdrcating to the user a priority for incoming communications on the basis of the leamt 
preferences. 



In accordance with the presem invention there is also provided a workstation 
including an input device operable by a user to control operation thereof, a visual detector 
for detectmg changes in visual environment at the workstation, and control device 
responsive to changes deteaed by the visual deteaor and to operation of the input device 
to provide an estimate of proximity of the user to the workstation. 

In order that the invention may be more fully understood, an embodiment thereof 
wJI now be described by way of example with reference to the accompanying 

drawings in which: 

Figure 1 is a schematic block diagram of a personal computer configured to aa as 
an mtelligent assistant device in accordance with the invention; 
Figure 2 is a schematic diagram of software modules used by the device- 
Figure 3 is a schematic diagram of the architecture of the co-ordinator shown in 

Figure 2; 
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Figure 4 is a schematic flow diagram illustrating execution of a plan determined by 
the co-ordinator; 

Figure 5 is a schematic view of the display provided by diary assistant shown in 
Figure 2, for making a diary entry; 
5 Figure 6 is a schematic view of the diary display; 

Figure 7 is a table of fuzzy rankings for the diary preference "early morning" shown 
in Figure 5; 

Figure 8 is a flow diagram of the diary entry process; 

Figure 9 is a block diagram illustrating operation of the e-mail assistant shown in 
JO Figure 2; 

Figure 10 is a flow diagram illustrating an analysis of the address data on incoming 
e-mail; 

Figure 11 is a block diagram illustrating a process for ranking e-mails according to 
the importance of their subject field to the user; 
15 Figure 12 is a block diagram of a process for monitoring the computer users past 
preferences for reading previously received e-mails; 

Figure 13 is a flow diagram of a process for monitoring the user's previous history 
of sending e-mails to particular addresses; 

Figure 14 illustrates schematically the display provided by the e-mail assistant, in 
20 accordance with the invention; 

Figure 15 is a flow diagram for controlling answering of incoming telephone calls; 

Figure 16 is a graph of the output of the gaze tracker shown in Figure 1; 

Figure 17 is a schematic flow diagram of a process performed by the multi-modal 

interface shown in Figure 2; 
25 Figure 18 is a schematic diagram of various user states detected by the process 

shown in Figure 17; and 

Figure 19 is a block diagram illustrating a process for ranking the stress level of the 
computer user. 

30 Overview 

Figure 1 illustrates a generally conventional personal computer workstation that 
comprises a conventional keyboard 1, a display screen 2 such a CRT or plasma 
display; a mouse 3, a processor 4, such as a Pentiimi™ processor, random access 
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memo^y 5. a hard disc n,emo,y 6. an audio input 7 such as a ™- U 
-erances from .he use. a .acW 8 .o detect ^ZZ 17 "^'^ 

and input/output interfaces 9 to connea the worJll T T "^^^ 
and .ider area networJcs (WANs) such as th toT 7^ 
^ exchange including e-ma;i ,««c • l " facihtate data 

All of the modules make u« nf w ^ co-ordinaior module 18. 

epnonedireaoiy service such as Talking Pages'- ..A u ^ • 

5 ages , as described m more 



detail in British Telecom Technical Journal (BTTJ) Vol. 16, No. 3, July 1998, "YPA 
- An Intelligent Director Enquiry Assistant" A. De Roeck et al pp 145-155. 

Module 15 comprises a web assistant for browsing the World Wide Web. The web 
assistant develops an interest model of the users interests and preferences, using 
intelligent reasoning techniques. This is described in more detail in British Telecom 
Technical Journal (BTTJ) Vol. 16, No. 3, July 1998, "The Personalisation of Agent 
Services" S. Soltysiak et al pp 110-117. 

Module 16 comprises an e-mail assistant the user to determined whether incoming 
e-mail warrants immediate attention or can be reviewed at a later date, using 
intelligent reasoning techniques. 

Module 17 comprises telephone assistant which determines whether a call should be 
answered immediately or safely handled at a later date. 

The various modules can thus be considered as assistants and will now be described 
in more detail. 

Ch<3rdinator 

The co-ordinator 18 operates as an intelligent scheduler to allow complex tasks to 
be carried out using more than one of the modules 11 to 17. The general 
architecture of the co-ordinator 18 is shown in Figure 3 and comprises a scheduler 
19, a plan library 20, an execution module 21 and a monitoring module 22. 

The scheduler 19 is operative to schedule the modules 11-17 of Figure 2 to carry 
out tasks individually, which together make up an overall plan. The modules 1 1 to 
17 are thus instructed by the co-ordinator 18 to operate in a particular sequence to 
perform instructed tasks which together make up an overall plan. 

A number of plans for the scheduler 19 are defined in the plan library 20. The 
execution module 21 dispatches instructions to the various modules 11 to 17 as 
specified by the scheduler 19, 



The monitoring n^odule 22 n^onitors the operation of the modules 1 1 to 17 and 
detects when instructions are received to perform a particular one of the plans. 

. The co-ordinator 18 may be implemented in Java™ utiHsing Java threads so as to 
enable the various funaions to be performed concurrently. 

An example of a multi-functional plan carried out by the co-ordinator wiU now be 
descnbedwuh reference to Figure 4. At step S4.1. the user makes an ent^. using 
- ^He diary assistant 11 (to be described in more detaillater). ITie user thus Les a 
diary entiy to arrange a holiday at a particular, pre-arranged time in the future 
Pnor to this diary entiy. the co-ordinator 18 will ensure that appropriate data is 
collated and entered into the diary so that the user can make a decision at the 
entered tune Thus, at step S4.1, the user is prompted to enter into the diary an 
mstniaion -holiday' which prompts the user for a desired holiday destinall. 

The monitoring module 22 of the co-ordinator 18. deteas the entry of the holiday 
diarisanon earned out at step S4.1. The detection is shown at step S4.2. Having 
detected that a holiday is to be arranged, the monitoring module selects at step S4.3 

20 a ^ I'^'^y Planfromtheplanlibraiy20showninFigure3.Theplanconsist ofa 
number of steps to be carried out by the various assistants in order to collate data 
tor the user relevant to arranging a holiday. 

At step S4^4. the scheduler 19 schedules a list of tasks to be performed at various 
umes by the assistants in order to obtain the data relevant to arranging the holiday 
In this case, the schedule involves the use of the classified directo^^ assistant 14 Id 
the web assistant 15. 



At step S4 5. the execution module 21 instructs the assistants 14. 15 to carry out 
tasks to obtain data relevant to the holiday. The directory assistant 14 is instructed 
to obtain data from travel agents listed in classified directories who offer deals 
relatmg to the desired holiday destination. The web assistant 15 is similarly 
.nstruaed to surf the web to find corresponding information on the internet Hius 
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the names and relephone numbers and possibly other information is obtained 
concerning travel agents who offer an appropriate service. 

At step S4-6, the monitoring module 22 receives the data obtained by the directory 
5 and web assistants 14, 15. At step S4.7, the execution module 21 sends the data to 
the diary entry slot in the diary operated by diary assistant 11. Thus, at the time 
dictated by the diary entry, the user is presented with suitable data to enable 
telephone calls to be made to travel agents in order to arrange the holiday. 

10 Thus, the co-ordinator 18 lifts the burden of many of the steps of arranging a 

holiday from the user by making reference to a previously defined plan and carrying 
out a schedule of tasks in accordance with the plan. 

It will be understood that a number of different plans can be held in the plan library 
25 20 to execute different tasks which may involve other ones of the assistants shown 
in Figure 2. A scheduler 19 allows the plan to be scheduled at different times so 
that the outcome of one task performed by one of the assistants can be considered 
when performing another task carried out subsequently by another one of the 
assistants. 

20 

When the task is completed, and the execution module sends the returned data to 
the diary entry (step S4.7). It may be that the diary indicates that the user is un- 
interuptable. This will be explained in more detail hereinafter. If this occurs, the 
co-ordinator 18 carries out a routine by which the data derived by carrying out the 
25 task is temporarily held and then rescheduled for display to the user at a different 
time. 

Diary Assistant 

The diary assistant 1 1 uses fuzzy reasoning techniques to assist the user in making a 
30 diary entry so that the user does not have to specifically select a particular diary slot. 

When the user selects the diary through the multi-modal interface 12, a diary entry 
screen is displayed on the display 2, as shown in Figure 5. The diary entry 
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details of the diarventrv;» ■ • ^ P^mpted to specify 

"1 i-nc oiaiy entiy I.e. narrative, in window 2^ TU:. ■ . . 

modal interface 12 e ^ • u . \ 

during an important meeting. ^ ''•S- 

Mouse cursor 26 can be used to enter a preferred start tin,e f . ^• 

window 27. '^'^^^'''^'"'"n'^fo'-thediaiyentiyin 

/J 

5 Lxus example, the user can ooerate rU^ i 

In the following description, it wiU be assumed that the . 

2^ has been selected and th.r ,1, I . . '^^^ window 29 

»ciectea and that the local mode is operative for j 
Figure 6. The earlv n,or„- j . Wednesday as shown in 

6 o. early morning wmdow 29 has associated fi.,^ 




It will be understood that the fuzzy rankings shown in Figure 7 are only one 
example of different values that can be used. Generally the values may be defined 
by a function which has a peak at the most desirable time and tails which decrease 
on either side, towards zero. For example a triangular function could be used, with 
5 a peak at the most preferred time and straight line slopes to zero on either side, with 
a total timewidth of two hours. Alternatively beU peak functions could be used to 
define the fuzzy rankings. 

Certain time slots may already be filled with previous diary entries and so the diary 
10 assistant seeks to find the most preferred free entry according to the fuzzy rankings. 
This will now be described with reference to Figure 8. 

At step S8.1, the assistant 11 reviews the time slots within the preference range set 
by the table shown in Figure 7. It will be understood that some of the time slots 

15 may have already been taken by previous diary entries. Then, at step S8.2, the 
assistant 1 1 attempts to make the diary entry according to the fuzzy rankings of 
Figure 7. If suitable time slots are available, this merely involves selecting the 
available time slots according to the order of preference shown in Figure 7, 
However, if sufficient time slots are not available, the assistant 11 makes use of an 

20 iterative improvement algorithm in order to review the previously entered diary 
items and their fuzzy rankings in order to determine whether they can be shunted, 
without changing their order so as to open a sufficient span of time slot to allow the 
new diary entry to be inserted. A suitable iterative improvement algorithm is 
described in "Artificial Intelligence - A Modern Approach" by S. Russell and P. 

25 Norvig, Prentice Hall, USA. 

At step S8.3, the result of rurming the algorithm is reviewed. If successful, the new 
diary entry is entered into appropriate time slots at step S8.4 and the previous diary 
entries are, if necessary shunted, without changing their order to accommodate the 
30 new diary entry. Alternatively, if the outcome of rurming the algorithm at step $8.2 
is unsuccessful, the diary entry is not made and is instead entered on a "to do" list 
at step S8.5. 
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It will be understood that the late morning and afternoon windows 30, 31 have 
associated fuzzy ranking tables that correspond to the table of Figure 7. but for the 
late morning and afternoon respectively. As regards the "around" window 32 a 
ftizzy ranking table is provided which defines ftizzy rankings in a time window 
^ around the entered time. Thus, the time slot which includes the entered time has 
the highest ftizzy ranking and time slots ftirther away from this have progressively 
lower fuzzy rankings. 

The previously described example constitutes a local mode of operation of the diary 
0 assistant, m which an ent^^ is made on a panicular day. However, in a global mode 
the ftizzy parameters may be used to find an appropriate slot in any available day 
Further search strategies can be devised. For example, a particular date range may 
be selected and other suitable restraints may be used as appropriate. Thus, the diaiy 
assistant facilitates the making of a diaiy without the specific need to look for a 
particular vacant diary slot. 

E-mail A ssistant 

E-mail has greatly facilitated communicarion interchange between users comiected 
m a network. However, e-mails tend to proliferate in number and a user at a 
particular workstation can be inundated with e-mails which take a significant 
amount of time to open and read. Some e-mails are of crucial importance whereas 
others are of only marginal interest. It would therefore be helpftd to filter e-mails 
according to priority. In some conventional systems, e-mails are given a priority by 
the sender but the user can only determine the actual priority by reading the e-mail 
Itself. The e-mail assistant 16 provides a solution to this problem. 

Referring to Figure 9. the e-maU assistant makes use of a Baysian net 33 in order to 
make a suggestion 34 as to whether the e-mail is of relatively high or low 
unportance and thus whether it should be read immediately or can be left until a 
later tmie. As known in the art. a Baysian net considers the causal relation of 
histoo' data in order to modify the a-priori probabihty of the occurrence of an 
event. A discussion of Baysian nets is given by S. Russell and P. Norvig, supra 
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The e-mail assistant provides inputs to the Baysian net 33 that comprise data 35 
concerning the address of incoming e-mails, data 36 concerning the importance to 
the user of the subject matter of the incoming e-mail, as signified by its title, and 
history data 37 concerning the user*s previous reading of e-mails for a particular 
sender and the user's previous sending of e-mails to the sender of the incoming e- 
mail. 

The derivation of data 35 concerning incoming e-mails will now be discussed in 
more detail with reference to Figure 10. As known in the art, e-mails can be 
addressed to an individual address, can be copied to one or more other people and 
can also be sent using a predefined mailing list, for example to all engineers in a 
particular company or to a group of customers. In this example, it is assumed that 
an e-mail addressed solely the user is of high importance. It is to assumed to be of 
lesser importance if copied to another person. If copied to more than one person, 
it is assumed to be of lesser importance still. If sent via a mailing list, it is assumed 
to be of even lower importance. 

Thus, referring to Figure 10, at step SI 0.1, the address information for the incoming 
e-mail is analysed to determine whether the e-mail is addressed solely to the user. If 
so, it is given a high importance ranking. A ranking factor is defined having a value 
range between 1 for high importance and 0 for no importance. In this example, the 
ranking is set equal to 1 when the incoming e-mail is directed to the user alone. 

At step S10.2, the address information is analysed to determine whether the e-mail 
is copied solely to another. If so, the e-mail is given a importance ranking = 0.8. 
At step S 10.3, a determination is made whether the e-mail is copied to several 
others. If so, the e-mail is ranked with an importance of 0.5. At step S10.4, it is 
determined whether the e-mail is part of a mailing list. If so, it importance ranking 
is set to be 0.3 i.e. low importance. 

It will be understood that the specific rankings shown in Figure 10 are examples and 
that in practice, the importance attributed to each particular category of e-mail by 
the individual user will change with time. The rankings can be thus learnt over a 
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period of time by momtoring the user's behaviour in regard to the reading of 
incoming e-mails. 

The data 36 concerning the subjea field of incoming e-mails will now be discussed 
^ m relation to Figure 11. As well known in the art, e-mails have as a header 

information concerning the sender, the recipient and the subject of the e-mail As 
shown m Figure 11. at step Sll.l. the subject of the incoming e-mail is detected 
from «s header. At step SI 1.2, a profile of important subjects is developed 
pertment to the individual user. For example, if a user has recently sent a number 
of e-mads usmg the same title entered into the subject field, the title of the 
mcommg e-mail is deemed to be imponant to the user. It will be understood that 
fuzzy recognition techniques can be used to identify different presentations of the 
same topic in the subject field. At step Sll.3. the importance of the incoming e- 
m^l .s ranked by making a comparison between the profile developed at step S 1 1 2 
and the actual subject of the incoming e-mail identified at step Sll.l. 

Deriv«ion of the history data 37 from previous e-mails wiU now be described It 
wdl be understood that in a conventional manner, the e-mail assistant keeps a log of 
e-ma.ls that have been sent and received with appropriate dates and times. Thus 
Che tune o receipt and the time of reading of a particular previously received e-mafl 
can be analysed in order to determine the time taken by the user to read a particular 
e-mad afier receipt. If the e-mail is read quickly after receipt, this indicates that the 
user considers e-maUs from a panicular sender to be important whereas if the user 
has m the past taken a long time to read an e-mail from a particular sender after 
receipt, this indicates that the user considers e-mails from a particular sender to be 
Ot low importance. 



n^us. referring to Figure 12. at step S12.1. an analysis is carried out of the time 
previously taken to read e-mails from individual addresses i.e. individual senders 
Then a. ,ep S12.2, when an e-mail from a particular sender is received, a decision 

made concerning the time taken to read e-mails from the sender, in the past. If e- 
mails previously were read within less than a given time, i.e. deemed to be 
u-ponan, the incoming e-mail is given a high ranking = 1. However, if previous e- 




mails from the sender were previously read in greater than the given time, the 
incoming e-mail is given a low ranking 0. 

This history data 37 also includes information concerning the history of sending e- 
5 mails by the user (operator of the workstation shown in Figure 1) to the sender of 
the incoming e-mail. This is shown in Figure 13. If an e-mail was recently sent by 
the user to the sender of the current, incoming e-mail i.e. the user is expecting a 
reply, the incoming e-mail is given a high ranking but otherwise a low ranking. 
Thus, at step S13.1, the address and time sent data on the e-mail log is analysed in 
20 order to determine when the user last sent an e-mail to the sender of the incoming 
e-mail. If an e-mail was sent within less than a given time, the incoming e-mail is, at 
step SI 3. 2, given a high ranking = 1 whereas if an e-mail was not sent within the 
given time, the incoming e-mail is given a low ranking 0. 

15 Referring back to Figure 9, the various data inputs to the Baysian net 33 are 

processed to provide the suggestion 34 concerning the relative importance of the 
incoming e-mail. It will be understood that not aU of the data inputs are 
necessarily needed for the net 33 to make a suggestion 34. An example of 
categories of suggestion 34 are set out in the following table: 



Table 1 


Suggestion 


Message 


High priority 


Suggest read mail now 


Mediimi priority 


Suggest read mail soon 


Low priority 


Suggest read mail later 



Figure 14 illustrates how the suggestion message 34 may be displayed to the user. 
The screen of display 2 (Figure 1) is shown with a work area 38 on which data 
25 processing is carried out. The status bar 39 is shown below the work area. Thus, 
the user may be operating a wordprocessor to prepare a document. When an 
incoming e-mail is received, a mail icon 40 is displayed on the status bar 39. This 
indicates the arrival of the e-mail but not its content. Thus, the user must decide 
whether to stop word processing and open the newly arrived e-mail or whether to 
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connnue word processing. If .he e-.ail is of peripheral i.portar,ce, it would be 
better to cont.nue with the word-processing rather than break concentration In 
accordance w„h the invention, the user n,oves the n,ouse cursor 26 onto the „.ail 
.con wbch results in a n,ail priority window 41 being displayed. The window 41 
^ cont.ns a message selected according to Table 1 above, depending on the 
suggestion made by the Baysian net 33 shown in Figure 9. 

.nformed of this fact and can continue word-processing. However, if the e-maflis 
'0 suggested to be of high priority, the user can stop word-processing and open the 
newlyrece.vede-n,ailn,essage. The message in window 41 is a suggestion onlv a d 
thus can be overridden by the user if deemed appropriate. ' 

Telephone assistant 

1^ Referring to Figure 15. a Baysian net 42 can be used to restria answering of 
.ncomang telephone calls. The net 42 is similar to the net 33 described with 
reference to Figure 9. Data is fed into the net 42 and an output 43 is provided to 
control answering of incoming telephone calls. TT^e call line identifier (CLI) of 
incoming calls is fed as data 44 to the Baysian net 42 

Also, the Baysian net 42 receives data 45 from the diary assistant 11. relating to 
diaiy entnes indicating that the user has planned certain aaivities relating to the 
aller i.e. the mitiator of the incoming call. For example, the diary may contain a 
diaiy indicating that a call is expected, in which case the net 42 tls tL into 
account when prioritising the incoming caU. Also, the diary may indicate that at a 
pani^ar time, the user is un-i„ten.ptable. in which case call answering w^l b! 



Funhermore. data 46 concerning the users prior histoiy of answering calls from a 
par^arCUandthemaldngofcaUsbytheusertothecallerisanatsed.J^^^^^^^ 

a.anl.ng.inasimilar manner to the way in which e-mails are analysed, as desc^^^^^^ 
relation to Figures 12 and 13. / . as aescnbed 
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Thus, the telephone assistant can instruct the interface 9 shown in Figure 1 to 
inhibit the ringing of a telephone so as to divert an incoming call to a messaging 
service, so as not to interrupt the user unless the incoming call is analysed to be of 
relatively high importance. Thus, relatively high importance calls are announced to 
5 the user and relatively low importance calls are not announced depending upon the 
analysis carried out by the Baysian net 42. 

Multi-modal interface 

As shown in Figure 1, the workstation includes a gaze tracker 8 which monitors 
10 movement of eyeball of the user. Thus, the direction of gaze of the user can be 

determined in order to allow the user to look at a particular part of the display 2 in 
order to control data processing operations by eye movement. One example of a 
gaze tracker is described in US Patent 4 836 670. Conventional gaze trackers 
consist of a TV camera directed towards the eye of the user together with pattern 
IS recognition software running on the computer, which analyses the imagine 

developed by the camera in order to determine the direction of view of the user. 
This is correlated with the configuration of the display so as to determine which 
window or region of the display is being viewed the by user. 

20 With a gaze tracker, it is difficuk to determine precisely when a user is present at 
the workstation. Figure 16 is a graph of the amplitude of signals detected by a gaze 
tracker's camera as a user firstly approaches the workstation, sits next to it ready for 
use and then leaves the vicinity of the workstation. The approach of the user 
produces an initial upward step in the output of the gaze tracker and then as the 

2S user becomes positioned next to the workstation ready for use, a pulse 47 is 

produced. Thereafter, the output settles to a relatively steady level. When the user 
leaves, pulse 48 is produced in the output. The level that occurs whilst the user is 
present, using the workstation is much the same as when no user is present at the 
workstation. It is therefore difficult to determine from the output whether the user 

30 is present or absent from the workstation. 

The multi-modal interface 12 includes intelligence in order to estimate the presence 
or absence of the user from the workstation. Referring to Figure 17, data input 49 
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15 



20 



compnse. data concerning the discontinuities or s^tches 47. 48 that occur when a 
use. enters and leaves the vicinit, of the workstation. Data input 50 concerns 
act..^ of the keyboard 1 and .ouse 3. When the keyboard or .ouse is found to 
be act., a flag KMB is set = active and when no activity occurs within a 
predetennu.edti„.,theflagissettoKMB = inaaive. 7T:e relationship of the 
status of flag KMB and switches in the camera output activit,is shown in F g^r e 18 
Estunator 51 shown in Figure 17 thus estimates the user activity on the basi Tthe 
o.c. d^agran, shown in Figure 18. In this way. it is possible t! esti.at : 
^ the user .s present, absent, or merely present but not active at the workstation. 

With this information, data fron, the camera of the gaze tracker 8 can be 
appropnatety processed on the basis of a realistic estin^ation of whether the user is 
actually present at the workstation thereby in^proving processing efficiency. 

2-ulti^odaI interface 12 .an also pro^de data to any one of the assistants 

step S19.1 the rate at which the user operates the "delete" key. Usage of the d . 

™ .nd^cates that the user is under high levels of stress. Sin^arly. heavy use of Z 
return key vindicates a high „ror rate and hence high stress This i 
;tep^.. AtstepS..3.thedatafro.stepsSl.. LsZ^Z:;::-!: 
torankthestressleveloftheuser. It will be underwood that the sysL can 
monitor the usage of the delete key and the return key under all conditions and thus 

etem^ne an average rate for the user so that when the usage rate exceeds the 
..erface 12. Other keys such as control keys can also be monitored to indicate 



o..o..pl.sthatarecurrent.be.gcarr,edout;;;;r^^^^ 
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of information supplied to the user in order to provide a relatively calm working 
environment. However, if the user is sitting at the workstation and does not 
demonstrate high stress i.e. bored due to lack of work, the co-ordinator 18 may be 
configured to only provide a very low level of filtering of incoming e-mails and 
telephone calls in order to keep the user occupied. 

Many modifications and variations fall within the scope of the invention, which is 
intended to cover all permutations and combinations of the individual modes of 
operation of the various assistants described herein. 
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Claims 



1. An intelligent assistant device for assisting in the management of infonnation flows for 
a user, comprising: 

a pluralitjr of infonnation processing modules to process individual types of 
infonnation flows for the user using intelligent reasoning computing techniques, and 

a coK)rdinator module to co-ordinate operation of the infonnation processing 
modules, the co-ordinator including a libraiy of plans corresponding to different tasks to 
be performed, a scheduler to schedule activities of the processing modules to perfonn die 
tasks, and an executor responsive to the scheduler to instrua the modules to perfonn die 
tasks according to one of the plans. 

2. An intelligent assistant device according to claim 1 wherein die infonnation 
processing modules are operable to process infonnation flows so as to provide time 



management, infomiarion management or communication 



management. 



3. An inteUigent assistant device according to claim 2 wherein die infonnation 
processing modules include at least some of a diaiy assistant, an e-mail assistant, a 
telephone assistant and a web assistant. 

4. An intelligent assistant device according to claim wherdn die scheduler is operable 
to cause die diaiy assistant to make a diar^ ent^^ in response to data received from anodier 

one of the assistants. 

5. An intelligent assistant device according to any preceding claim wherein die 
infonnation processing modules include a multi-modal interface for user inputs of a 
plurality of different modalities. 

6. An intelligent assistant device according to claim 5 whereb die multi-modal 
interface provides the co-ordinator widi data concerning die level of str^ experienced by 
the user. 
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7. An intelligent assistant device according to any preceding claim wherein the co- 
ordinator module includes a monitoring module to monitor infomiation flows in the 
device. 

8. A computer program produa comprising a storage medium in which is stored 
computer readable code corresponding to an assistant device according to any preceding 
claim. 

9. A computer program produa comprising a storage mediimi in which is stored 
computer readable code conresponding to a co-ordinator as defined in any preceding dainL 

10. A diary module for an intelligent assistant device, including means to define a 
plurality of timeslots, input means to provide a plurality of predetermined fiizzy definitions 
of a desired time for a diary entry, means defining a relationship for each of the fuz2y 
definitions corresponding to timeslots for the entry ranked in an order of preference, and 
processing means responsive to a diary entry from the input means to review the timeslots 
and allocate the entry in accordance with the ranked order of preference. 

11. A diary module according to claim 10 wherein the timeslots define periods in a day 
and the fuzzy definitions cover respective different sectors of the day. 

12. A diary module according to claim 1 1 wherein the different sectors include 
moming and afternoon. 

13. A diary module according to daim 10 wherein the input means permit the entry of 
a desired time for the diary entry and the relationship for the fuz^ definition defines 
timeslots for the entry ranked in an order of preference relative to the entered time. 

14. A diary module according to daim 10 induding means for selecting whether the 
diary entry can be made over a restricted or imrestricted range of timeslots. 

15. A diary module according to daim 14 wherein said range of timeslots extends over 
a plurality of days. 
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16. 



A diary module according to any one of claims 10 to 15 wherein the 



processing 



means is operable to shunt previously entered diaiy entnes to accommodate a new entiy 
such as to accord T^th the preferences of aU the entries. 

s 

17. A computer program produa comprising a storage medium in which is stored 
computer readable code co^esponding to a diary module according to any one of claims 10 

to 16. 

^0 18. A communications organiser for communications made over a network, 

compHs^gmeans for modtoringcommudcarions received byausertogether^det^^ 
of the sender, leading means to learn the user's preference for communications from 

pamcular senders, and means for indicating to the user a priority for incoming 

communications on the basis of the learnt preferences. 

IS 

19. Ano,Kani^raccordingtoclaiml8whereinthecommunicationscomprisee.mails 
Ili'!'!^!"""' " '° '° ' -oniing 



messages. 



20 20. 



An organiser accordbg to claim 18 or 19 wherein the lear^g ^.eans includes 
Bayes net. 



2$ 



21. An organiser according to any one of claims 18 to 20 wherein the leaniing 
responsive to die subjea matter of the incoming communication. 



30 



22. An organiser according to claim 21 including means for monitoring keyboa«l 
operation eiror rate to determine the stress level measurement. 

23. An or^er accorxling to any one of daims 18 to 22 wherein the communications 

compnse telephone caUs and the leanungmeans is operable to permit anno^^^^ 
incoming call in dependence on said priority. 
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24. An organiser according to any one of claims 18 to 23 wherein the learning means is 
responsive to history data concerning the user*s making of communications to the sender 

25. An organiser according to any one of claims 18 to 24 wherein the learning means is 
5 responsive to histoiy data concerning the iiser*s receiving of communications from the 

sender. 



26. A computer program product comprisii^ a storage medium in which is stored 
computer readable code con^sponding to an assistant device according to any one of 
10 claims 18 to 25. 



27. A workstation including an input device operable by a user to control operation 
thereof, a visual deteaor for detecting changes in visual environment at the workstation, 
and control device responsive to changes detected by the visual detector and to operation 

1$ of the input device to provide an estimate of proximity of the user to the woikstation. 

28. A workstation according to claim 27 wherein the visual detector comprises a 
camera for a gaze tracker. 

20 29. A workstation according to claim 28 wherein the control means is operable to 
detect discontinuities in the output of the camera. 

30. A workstation according to any one of daims 27 to 29 wherein the input device 
comprises a manual input device. 

31. A workstation according to claim 30 wherein the manual input device includes a 
keyboard. 

32. A workstation according to claim 30 wherein the manual input device includes a 
30 mouse. 



33 . A workstation according to claim 3 1 including means responsive to keyboard 
activity to determine stress levels of the user. 
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34. A computer program produa comprising a storage medium in which is stored 
computer readable code corresponding to control means as defined in any one of claims 17 



to 33. 



35. An inteUigent assistant device substantially as hereinbefore described with reference 
to the accompany drawings. 

36. A diaiy module substantially as hereinbefore described with reference to the 
accompany drawings. 

37. A communications organiser substantially as hereinbefore described with reference 
to the accompany drawings. 

38. A woricstation substantiaUy as hereinbefore described with reference to the 
accompany drawings. 
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